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During the 1968-69 school year, the Southwest Regional Laboratory 
carried out a study to determine requirements for the installation of 
the First— Year Communication Skills Program, an obj actives— based kinder— 
garten. reading program. The purpose of the study was to identify the 
elements necessary for a full-scale installation of the Program during 
the 1969-70 school year. This report presents a summary of the infor- 
mation acquired during the study. Included is a description of the 
objectives , materials, procedures, sample, and results. The management, 
training , and evaluation systems related to the installation 
Program are also specified and defined. 
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INSTALLATION requirements for the swrl first-year communication skills 
PROGRAM: EVALUATION DATA, 1968-69 



Paul E. Rasta and Ralph A. Hanson 

Program installation as a specific task and focus of research In 
the developmental cycle of instructional programs is a process which 
has received little attention in the past. The responsibility of the 
curriculum developer has traditionally ended with the production of 
the educational materials and instructional guides for the teacher. 

As a result, many curriculum projects which proved quite successful 
in controlled developmental contexts have failed to achieve these same 
high levels of performance when In the natural classroom environment. 

Aware of the possible pitfalls of such a limited developmental proce- 
dure, the Southwest Regional Laboratory for Educational Research and 
Development (SWRL) undertook the task of extending the research and 
development effort beyond the production facilities and directly Into 
the schools. 

During the 1968—69 school year, SWRL carried out a study to determine 
requirements for the installation of the First— Year Communication Skills 
Program ( FY G S F ) ——an object i ve s — b as eel kindergarten reading program. The 
study was to (1) describe the levels of program performance and accep- 
tance obtained using the limited installation procedures devised prior 
to the study and (2) identify the elements necessary for a full-scale 
installation of the Program during the 1969—70 school year. This report 
presents a summary of the procedures followed and information acquired 
during the study. Included is a description of the study objectives, 
materials, procedures, sample, and results. 

First-Year Communication Skills Prog ram. The FYOSP includes a 
series of research— based instructional materials and procedures for 
teacher use in developing reading competencies in kindergarten chil- 
dren. It underwent extensive classroom testing before being used in 
the schools, including a product tryout involving 18 kindergarten 
classes from 10 school districts in the Pacific Southwest during the 
1967-68 academic year. 

The Program is organized into 10 relatively equal units of instruction 
(32 weeks). Upon completion of the Program, children should be able to: 

1. read all 100 Program words; 

2, read initial and ending word sounds; 

3. blend initial and ending sounds to form new words; 

4, name all letters of the alphabet. 
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The general Instructional procedures employed by the teacher are (a) to 
introduce the sequential activities in a given unit, usually covering 
a 3-week period; (b) to administer a Criterion Exercise (or test) covering 
each unit; and (c) to provide practice exercises for those who do not 
achieve mastery (a score of 16 or better on the 20-item Criterion 
else)* This procedure is followed for each of the 10 units. The entire 
Program is designed to operate within a 30-week instructional period, 
requiring approximately 30 minutes of daily instruction. 



Study Objectives 



Prior to conducting the study, an attempt was made to define the 
major tasks and initial procedures for program installation. Three 
major areas were identified for the installation of an instructional 
program, including: management, training, and evaluation. The specific 

functions and study objectives in each of these areas are defined as 
follows , 



Man ag ement 

The management procedures of a program include the functions, 
personnel, and materials required to manage the prescribed pupil 
instruction in the classroom and to monitor performance across class— 
rooms and schools within a district. Prior to the installation of an 
instructional program, identification and implementation of initial 
procedures and personnel resources to be used with the instructional 
program must be made. Also, provisions for the subsequent modification 
or deletion of ineffective procedures should be considered. 



Training 

The instructional program and accompanying management procedures 
require that school personnel carry out specified tasks. The training 
procedures provide these personnel with the requisite skills necessary 
for executing these tasks. To develop these training procedures initial 
specifications must be obtained on the: 

1. personnel skills and tasks necessary to install the instructional 
product ; 

2. training requirements for teachers and other school personnel 
responsible for implementation of the program; 

3. installation techniques and materials that will produce reliable 
training effects in the natural classroom environment without 
the supervision or assistance of external (e,g., SWRL) personnel. 
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Evaluation 



The evaluation procedures serve two major functions in program 
installation- The first is to provide information on the effectiveness 
of the training and management procedures devised for use with the 
program . These data will be used in a formative evaluation context 
to aid in the revision and development of procedures for future instal- 
lation * To fulfill this function * it is necessary to: 



L obtain a clear definition of the objectives specified for each 
of the training and management procedures; 

2. specify the sources of available data relevant to each procedure; 

3 , develop adequate instrumentation and data gathering procedures 
to obtain the data; 



4. devise a system of storage and analysis for quickly and 
accurately summarizing the data gathered. 



The second function of evaluation is to provide estimates of the overall 
program effectiveness in the classroom settings where the program may 
eventually be used. The summative evaluation data provide baseline 
information on the level of performance to be expected for an installed 
program . 



Study Procedures 



The basic study strategy was to prepare an initial set of training, 
management , and evaluation procedures for the FYCS Program, These 
procedures j along with the instructional program, would provide a 
vehicle for gathering tryout data on the problems of program installa- 
tion. 

Training Procedures and Materials 

The tryout of the FYCS Program required that some training be 
provided to personnel who were to install and implement the instructional 
program- To develop training procedures, minimum personnel skills and 
tasks necessary to use and manage the instructional program were 
identified. Based on this analysis, training materials and informal 
training sessions were planned. Primary reliance for effective use 
of the FYCS Program was placed on the SWRL Teacher’s Manual and brief 
meetings with selected district personnel. 
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Tea che r 1 s Manual 

A Teacher 1 s Manual (1969) containing the rationale ^ content s organic 
nation and procedures of the FYCS Program was developed by the SWRL staff 
A copy of this manual is included in the Instructional Program materials* 



Supervisor Orientation Session 

A one-day orientation session was planned for district personnel 
assigned responsibility for implementing the program within their 
respective districts* The SWRL staff presented five supervisors, one 
from each district, with the rationale, content, organization, procedures 
and monitoring requirements of the program. 

These training procedures allowed the Laboratory to assess the 
validity of two commonly -he Id assumptions made in relation to school 
use of instructional programs. The first assumption was that the only 
requirement for effective teacher use of an instructional program is a 
comprehensive Teacher’s Manual, Consistent with this assumption, many 
publishers consider all training requirements fulfilled with the delivery 
of a Teacher’s Manual, 

The second assumption was that additional teacher training require- 
ments (e.g,, in a complex instructional system) can best be satisfied by 
providing a few district personnel with a verbal "walk-through 11 that 
gives the rationale of the program. It is assimed that these "trained" 
personnel will s in turn, be able to adequately train the teachers in 
their districts. 



Management Procedures and Materials 

Preparations for the tryout included specifying simple procedures 
by which the districts and schools involved could monitor its use. 

These began with district personnel conducting training sessions for 
all teachers. Further, district personnel were asked to contact teachers 
using the program within their districts on a regular basis throughout 
the school year. Finally, district personnel were asked to act as a 
liaison in monitoring the collection of data on the program* 

The procedures developed all relate to program management within a 
district rather than instructional management in the classroom. Pro- 
cedures for the teacher to follow in managing instruction were available 
as part of the regular instructional sequence. These procedures had 
been verified as effective in previous small-scale tryouts and all were 
specified in the training materials for teachers. 
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EvaXuation ProcedurGs and Sources 

Provisions were made to collect information for both formative 
and summative evaluation purposes* The data for formative evaluation 
were collected on the management and training procedures 5 and data 
for summative evaluation were collected on program performance and 
-mccep t an ce * 

The nature of the data collected from persons involved in the 
tryout is described below* Groups involved include district supervisors, 
classroom teachers, pupils, their parents, and the school principals* 



1 , Dis trict Supervis ors i 

a. amount of orientation and training provided teachers on 
the FYC3P 

b. use and value of training materials provided 

c. amount of contact and interaction with teachers 

d* dissemination of information about the Program to district 
personnel and parents 

e, amount of time required to implement the Program 

f , specific problems with the Program which were brought to 
their attention 

g, suggested additions and revisions to Program training 
materials or procedures 

2, Classroom Teachers: 

a* amount of time spent on each instructional activity and 
record keeping 

b * use of the instructional methods 

c. demographic characteristics of pupils 

d. Program acceptance and suggested revision to procedures 

a, amount and sources of assistance given regarding the 
Program 

f, expected levels of pupil performance 

g, reporting of pupil progress on Criterion Exercises 
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3, Participating Pu, ils : 

a. performance information on the Criterion Exercises 

b . extent of pupil acceptance of activities 
4 . Parents of Pupils i 

a. information on parental acceptance of the Program 

b. parental participation in the Program with the child 
outside the classroom 

5, School Principals: 

a. Program evaluation 

b, distribution of materials 

c= Information provided parent, other schools, about the 
Program 



Data Acquisition and Instrumentation 

A number of data gathering procedures and instruments were developed 
to obtain Program information from the data sources. A special effort 
was made to minimize the record-keeping and data-acquisition tasks of 
the teachers. The following instruments were used to acquire the data: 

The Pupil Data Form . A class roster containing the name, sex, age, 
Entry” Skills Test score, language statistics, previous education, hearing 
vision, and speech problems of each student in a given class. The form 
was completed by each teacher and submitted to SWRL, 

The Weekly Activity Log. The Log was designed to monitor class 
progress in the Program by reporting the number of minutes per day spent 
on Program-related activities, the current unit and activity numbers, 
and the number of children individually assessed. Each copy of the 
form was printed in triplicate. Teachers completed the form weekly , 
retained one copy, and submitted the others to their Program supervisors 
and SWRL. SWRL was notified of new students entering the Program by 
the teacher's notation at the bottom of the Weekly Log. 

The Class Record Sheet , The record sheet was designed to be used 
by the teacher for recording pupil scores following a Criterion Exercise. 
On this form the teacher entered each pupil's name, his scores on each 
of the four Program outcomes and his total score. Using this form, the 
teacher was able to easily identify pupils requiring additional practice 
on specific Program outcomes. This form was also prepared in triplicate 
with copies to the teacher, the supervisor, and SWRL. 
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Questionnaires , Questionnaires were developed primarily for 
Program evaluation id revision as well as to provide additional data 
on school, classroom, and pupil characteristics. They were sent to 
teachers, principals, and supervisors after completion of the first 
unit of the Program and again at the end of the Program, 

Teacher Observation Scale . A classroom observational study was 
conducted to determine the extent to which participating teachers 
demonstrate desired instructional behavior. Teachers’ Instructional 
behavior in the classroom environment was observed, i^ecorded, and 
analyzed . The results of the study were then used for deriving teacher 
training objectives and developing classroom monitoring instruments 
to be used by Program supervisors. Two observation scales were used 
for recording the instructional interaction be tween the teacher and 
the pupils: the Instructional St imulus— Response Observation Scale and 

the Response Content and Dispersion Observation Form, The first scale 
was to provide data for a stimulus-response analysis of the instructional 
transactions between the teacher and a group of pupils. The Response 
Content and Dispersion Observation Form was to provide a measure of the 
patterning and dispersion of pup 4 l responses and the extent to which 
pupils are presented stimuli appropriate to the Program outcomes. 

Activity Preference Form , The Form was developed to determine 
differences in pupil attitudes toward Program and non— Program instruction. 
The Form consisted of one practice picture and five stimulus pictures 
(depicting different Program and non— Program activities) paired in all 
possible ways to make a total of 10 pairs. Upon presentation of each 
pair of pictures (accompanied by a verbal description by the examiner) 
the pupil was required to mark the bubble by the picture "he would like 
to be in most,” The Form was Individually administered to a sample of 
pupils by SWRL staff members. 

Parent Questionnaire . This questionnaire was designed to measure 
the attitudes of participating parents toward the Program and kinder- 
garten reading in general. It was sent to a sample of parents in one 
district . 

Each of these measures was thoroughly reviewed and pilot tested on 
small groups at SWRL before being used to gather information in the 
Study , 

Test Administration ■ Administration of tests to kindergarten pupils 
by SWRL personnel required careful coordination with the schools . Back- 
up plans were developed in the event of nonavailability of a specific 
sample of class pupils on the prearranged day* Schools and classes were 
contacted well in advance of the testing day and were informed of the 
purpose of each testing session. 
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Data Retrieval, Processing, and Storage 

Static Pupil Data . As each teacher returned her list of pupils 
on the Pupil Data Form, EWRL compiled a Master List, on which each 
pupil was assigned a permanent class and pupil identification number. 

The pupil information was then prepared for computer input. Retrieval 
letters were sent to teachers who did not meet the October 25 deadline. 

Weekly Activity Log., As each teacher’s Weekly Log was received, ^ 
it was checked off on a Master Chart. The data were then precoded and 
prepared for computer input. When logs were not received, retrieval 
letters were sent to tlie district supervisors*. 

Class Record Sheets . As the Class Record Sheets were received, 
they were recorded on a Master Check-off list. The data were precoded 
and prepared for computer input. If a student’s scores were missing 
due to absence, a retrieval letter was sent to the teacher. 

Program Evalua tions . Received Program Evaluations were recorded 
on a Master Check-off list and tabulated by hand, 

A master schedule for the collection of all data was developed 
detailing the interrelationships between samples, Instruments, and 
dates for the collection of information. 

The computer was utilized to handle all the data collected with 
the exception of the data provided by questionnaires. Manual forms 
designed to facilitate direct transcription to punched cards were 
used ; however, coding sheets were required for some forms to minimize 
errors and reduce keypunch time. All forms received were screened; 
those containing poor handwriting, confusing sentences, etc, , were 
precoded. After the cards were punched and verified, they were loaded 
onto the computer in the appropriate data base. 

Two data bases were developed to monitor pupil and class performance 
during the tryout. The first stored performance and pupil demographic 
information while the second stored weekly log information. 

The data bases were updated when a new "batch" of approximately 
3000 cards was ready to be merged or added to data already entered on 
the computer. The Weekly Log form required four data cards per form 
sheet while the Criterion Exercises required five data cards per form 
sheet. 

The steps followed in processing manually collected data were to: 

1, precode raw data 

2, enter data on code sheets 

3, rough-check code sheets 
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4. keypunch and verify cards 



5. generate printout 



6, proof printout; correcting keypunch errors 



7 * merge into the appropriate data base 



8. generate printout; proof and correct 



9. generate storage tape 



10. enter data on computer 



11. utilise data base for specific analyses 



The Study Sample 



Selection Criteria 

The sample included approximately 2,100 children from ?6 schools 
in five urban districts in three states. 

The criteria for sample selection were: (1) that the Program be 

used on a district-wide basis, (i.e., at least two or more schools), 
(2) districts be distributed across the region, and (3) each district 
provide a coordinator to carry out the orientation and training of 
teachers , materials acquisition and monitoring of the program. 

Schools Included 

A sample of schools was selected from each of the five school 
districts. The selection was made by the district office rtaff in 
each case and was based on factors such as: 

1. principal and teacher interest in Program 

2. number and type of existing experimental programs in the 
school 

3. teacher work load 

4. session length 

All kindergarten classes in the selected schools participated in 
the Program. The total number of schools, classes, teachers, and 
pupils from each district is shown in Table 1. 
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Table 1 

Number of Schools, Teachers, Classes, 
and Pupils Per School District 




Pupil Characteristics 

The characteristics noted for analysis were: 

1, age: in years at the time of Program tryout 

2, sex: male or female 

3, preschool education: experience in any formal instruction 

or education (e.g,, Head Start, nursery school, church program) 

4, languages spoken: ability to speak more than one language was 

described as multilingual 

5, sensory and speech impairment: the presence of visual, hearing, 

or speech Impairment 

6, prior education: kindergarten repeats 

7, entry skills: based on performance of students on the SWRL 

Entry Skills Test 

Classification of children in characteristic 5 was based on either 
physical examinations by school medical personnel or teacher judgments . 
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The distribution of the children in the various categories at 
the beginning of the Program is shown by district in Table 2. 

The pupils tended to be quite similar with respect to the age and 
sex distributions in all participating districts. There was some 
variability from district to district in the number with previous 
education, the number who spoke more than one language, and in the 
incidence of visual, hearing, or speech impairment. 



Table 2 

Descriptive Data in Percentage for 



Pupils 


from Participating 


Districts 








Data 


District 

1 


District 

2 


District 

3 


District 

4 


District 

5 


Boys 


55.9 


48.9 


49.1 


46.1 


49.4 


Bilingual 


4.3 


7.4 


2.8 


4.0 


7.7 


Having Sensory Problems 


2,5 


3.0 


3,5 


1.3 


5.3 


Having Speech Problems 


5 ,1 


7.4 


2.8 


4.0 


6.2 


Involved in Previous 
Education 


9,0 


23.7 


16,9 


24.5 


30.9 


Born Prior to 1963 
(the older pupils) 


12.0 


11.9 


11.2 


17,1 


12.9 


Mean Age in Years 


5.3 


5.4 


5.3 


5.4 


5.3 


Mode Age in Years 


5,0 


5.3 


5.2 


5,1 


5.2 


Total Number of Pupils 


553 


135 


509 


375 


599 



A chi— square technique was used to test the hypothesis that the 
proportion of pupils in each category was the same for all districts. 
The results of this analysis are summarized in Table 3, The chi-square 
values for previous educational experience and bilingualism were 
significant at the one percent level, indicating that the hypothesis 
of equal proportions could not be accepted. In summary, the samples 
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frotn the five districts appear to represent the same population with 
respect to age, sex, and sensory and speech problems , but not with 
respect to bilingualism or previous educational experiences. 

Pupils f Entry Behavior 

Participating pupils were further described in terms of specific 
pertinent skills before beginning the Program. The skills were measured 
by a 20-itetn "Entry Skills Test" administered to pupils in the five 
districts just before the Program was introduced. A score of 18 or 
more correct was established as the minimum level required for Program 
participation. Mean scores of pupils in each district and percentage 
scoring 18 or more are given in Table 4. Over 94 percent of the pupils 
in all districts obtained scores of 18 or greater on this 20-item test. 



Overall Performance Characteristics of the Program 

The primary measures used to assess pupil performance were (1) the 
Criterion Exercises (administered by the teacher to all students at the 
end of each unit) ; and (2) the final Program test (given to a sample 
of students from classes in two of the participating districts) . The 
correlations between the Criterion Exercises and the final test scores 
were very high (r=. 80) despite their restricted variance. Therefore, 
the results of the Criterion Exercises are emphasized in this section 
since these data are available for all pupils in the Program. 

Table 5 gives the means, standard deviations, and the percentage 
of pupils scoring 16 or greater on each of the Criterion Exercises. A 
graphic representation of the percentage of pupils scoring 16 or greater 
over the 10 units is given in Figure 1. The means for all 10 units are 
above 16, with the lowest performance levels attained on Units 3 and 4. 

Table 5 shows that the number of pupils taking each unit declines 
progressively from Unit 1 through 10. To determine performance charac- 
teristics of pupils who completed the entire Program, a separate table 
was prepared. The data on this subgroup are given in Table 6. 

Students who progress through all 10 units performed at a slightly 
higher mean level with a greater percentage scoring above 16 than the 
total group. This is illustrated graphically in Figure 2 where the mean 
scores for the two groups are plotted. 

Across all 10 units, the composite percentage reaching criterion 
(16 or greater) was 84.1, clearly exceeding the prespecified Program 
objective. 



Program Performance 
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Table 4 

Mean Score on Entry Skills Test 
Per District 



District 


Number of Pupils 
Taking Tes t 


Mean Score 


Percentage of Pupil 
Scores Greater Than 18 


1 


301 


19.399 


93.5 


2 


131 


19.310 


96.1 


3 


443 


19.413 


93.1 


4 


328 


19.643 


96. S 


5 


501 


19 . 396 


92.3 


Total 


1704 


19 .432 


94.4 
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Eig. 1. Percentage of pupils reaching 80% level on Criterion Exercises. 
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Summary data on pupil performance on each of the Program outcomes 
are given in Table 7. In addition, the table gives the mean total 
score for each of the 10 criterion tests* These data indicate that 
the lower mean performance on Criterion Exercises 3 and 4 is largely 
attributable to the lower pupil performance on Outcomes 3 (ending 
sounds) and 4 (blending) for these two units. 

Program Performance by District 

The summary performance information for each of the five participating 
districts is given in Table 8# Differences In performance between districts 
are due, largely, to the mean difference attained on the ending sounds and 
blending outcomes (3 and 4) # This can be seen clearly in Tables 8 through 
13 s which give the performance data by units for each district. 

A partial explanation for this performance differential is suggested 
by the time data provided* Districts 4 and 5 spent more total time on 
the first five instructional units and did not proceed into the next 
unit until high levels of mastery had been obtained. This suggests that 
spending more time at the outset of the Program may facilitate mastery 
of skills in later Instructional units. For the 1969—70 tryout } assess- 
ment procedures have been developed which provide the teacher with feed— 
back on pupil performance each activity. Through use of these 
procedures, instruction will be paced to provide performance levels 
which are consistently high across all units and outcomes. 

Evaluation of Mid— and End*-of— Program Test Results 

Mid— program and final tests were designed to measure pupil perfor- 
mance on the Program outcomes covered in units 1—4 and 1—10, respectively. 
These tests included both constructed and selected response items and 
were individually administered to pupils by SWRL personnel. The mid- 
program test was given after completion of unit 4 to a sample of 
pupils in District 5* The final test was given after unit 10 to 
pupils in Districts 5 and 2. 

These data provided information on two important questions : 

1. What is the relationship between pupil performance on the 
mid— program and final tests and achievement on the Criterion 
Exercises ? 

2. What level of performance on the unit Criterion Exercises 
(selected responses) is required for attainment of the 80% 
level of mastery on the constructed response Items of the 
mid— program and/or final test? 

Table 14 gives the performance of pupils on the mid— program and 
final tests compared to their average Criterion Exercise Scores. 
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Summary Data on Pupil Performance 
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Table 14 shows a high correlation between test scores (r=* . 80) . The 
percentage of pupils reaching an 80% criterion level on the constructed 
response portion of the mid-program and final tests, however, was 
markedly less than pupils 1 corresponding performance on the Unit 
Criterion Exercises * The constructed response items were more 
difficult than the selected response items. 

In order to see what performance level on the Criterion Exercises 
would be needed to reach an 80% criterion level on the mid-progx~am and 
final tests, scatterplots of the two score sets were constructed. These 
scatterplots are given in Figures 3 and 4. 

Inspection of the data reveals that: 

1. Within each column representing an average Criterion Exercise 
score, there Is much variance, e.g., some pupils averaging 

20 on the Cr’iterion Exercises scored less than 70% on the 
mid-program test. 

2. Until an average Criterion Exercise score of 18 was attained, 
there was almost no chance that a pupil would score above 
80% on the constructed response test. 

3 . More than half of those pupils whose average Criterion 
Exercise score was 18 or 19, scored less than 80% on the 
mid-program test. 

Based on these results, it was suggested that the Program Criterion 
level on Criterion Exercises be raised from the 80% (16 correct) to the 
90% (18 correct) level of performance. This suggestion has been adopted 
for the 1969-70 tryout. 



Comparison of Morning and Afternoon Classes 

An approximately equal number of pupils were in the morning and 
afternoon groups. The performance of pupils in morning versus afternoon 
sessions is given in Table 15. A slight difference favoring the children 
attending the afternoon session can be noted across all units. Once 
again, the differences seem to diminish as one moves from unit 1 through 
unit 10. Part of this difference may be due to improved instruction 
given to the afternoon group due to the teacher’s Increased familiarity 
with the materials and procedures after teaching the morning session. 

Pupil Characteristics and Performance 

Information was obtained on a variety of pupil characteristics. 

The relationship between their characteristics and performance on the 
unit Criterion Exercises is presented in this section of the report. 





Cons true te d-Respons e Score- -Mid-Program. Test 
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Fig. 3. Correlation and regression analysis o£ relationship between 
average criterion exercise scores and scores on constructed 
response items on m i d « p r ogr am examination. 




Criterion Exercise Average (Units 1-4) 








Constructed Response Scores 
Y = 18.396 y = 5.592 
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Fig. 4, Correlation and regression analysis of relationship between 
average criterion exercise scores and scores on constructed 
response items on final examination. 

(for 159 District 5 pupils) 



r = .80 

p < .001 




Average Criterion Exercise Score 
X = 17.830 x = 2.055 
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Pupils in Morning and Afternoon Kindergarten Sessions Reaching Criterion 

Across All 10 Units of the Program 
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Entry Skills . One criterion established to determine readiness 
to participate in the SWRL Program was the Entry Skills Test. Pupils 
scoring 18 or above were considered ready to participate in the Program. 
Only 6% of the participating pupils scored less than 18, Table 16 
gives the mean score and percentage of pupils scoring 16 or better on 
the unit tests for those pupils above and below the criterion of 16 on 
the Entry Skills Test, The data indicate that the Entry Skills Test 
is a valid indicator of pupils who will not reach criterion on the 
unit Criterion Exercises , especially on the early units in the Program, 
Note that the difference between the two groups on criterion performance 
diminishes across the 10 units. 

To Illustrate 5 of those pupils who progress through unit 10* 80% 
who had scored below criterion on the Entry Skills Test reached criterion 
on unit 10, Only 48% of these same pupils reached criterion on unit 1. 

Bilingual-Monolingual . Approximately 5% of the total tryout sample 
was bilingual. The pupils were classified bilingual if their primary 
language was other than English, Spanish was the first language for 
the majority of these children. Their performance is compared with that 
of the monolingual children in Table 17* The results indicate that 
monolingual children (English only) performed at a higher level on the 
criterion tests across ail ten units. Further* the relative difference 
appears to remain approximately the same through all phases of instruc- 
tion , 

Sex Differences in Perfor m ance , Table 18 indicates that females 
exceeded males across all ten units on mean score performance as well 
as in percentage above criterion - 

Age , Table 19 shows the differences in criterion test performance 
for younger and older children. Younger children were defined as less 
than 5 years and 3 months of age as of September* 1968, The performance 
of the younger group lags behind that of the older throughout* although 
the difference tends to decrease across the units. Indeed* for unit 10* 
the percentage reaching criterion for the younger group actually exceeds 
that of the older group and mean score differences are negligible. 

Sensory and Speech Impairment , Children with visual* hearing, and 
speech impairments were identified by teachers and school personnel at 
the outset of the Program. These children represented a very small 
percentage of the total tryout population. Interestingly enough* com- 
parison of the performance data indicates that children with visual and 
hearing deficiencies performed better than children without such 
deficiencies, with the differential increasing from unit one throughout 
ten (Table 20) . Children with some speech impairment scored slightly 
below the children without impairment on the early units* but they 
scored evenly with the unimpaired group on later units. 
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Pupils Scoring Above Criterion (16 or More) on All Units 
Who Scored Above and Below 18 on the Entry Shills Test 
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Table 19 

Pupils Scoring Above Criterion (16 or More) on All Units for Younger and Older Child 
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Pupils with Visual, Hearing, and Speech Impairments 
Scoring Above Criterion (16 or More) for All Units 
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Kxnd er ga y ten Re peat: bits . A small proportion of the participating 
pupils (3%) were kindergarten repeaters. The results of their perfor- 
mance on the Program Criterion Exercises is given in Table 21, along 
with the overall performance of Program participants. Their perfor- 
mance was slightly above that of the other pupils at the outset but 
fell behind on later units. 



Preschool Education . Approximately 19% of the students participating 
in the tryout entered with some form of preschool education. These 
children were placed in one of three categories, i,e. s nursery. Head 
Start, or church school. The majority of the children (11% of the 
total population) with prior education received it in some form of 
nursery school. Throughout they performed at a higher level than 
pupils with other forms of previous education or no previous education. 
Similarly, children who had church school experience exceeded the 
children with a Head Start education or no previous education. Head 
Start pupils performed at a lower level across all units than pupils 
in any of the other categories. These data are presented in Table 22. 



Program Acceptance 



A critical aspect in the evaluation of any research-based instructions 
program is that of user acceptance. Regardless of the demonstrated 
effectiveness of the Program, it may not be accepted by the user popu- 
lation. To obtain information on the degree of user acceptance of the 
Program, the following questionnaires were developed by SWRL , using 
scaled and open-ended items : 



Teacher Evaluation Form 
Teacher Final Questionnaire 
Principal Evaluation Form 
Principal Final Questionnaire 
Supervisor ’I^aluation Form 
Supervisor Final Quest ioma ire 
Parent Questionnaire 
Field Visits 

Parent-Assisted Learning Questionnaire 

Non- Par tlcipan t Form 

Pupil Activity Preference Form 

An analysis of the questionnaire attitudinal data revealed a 
favorable user response to the Program. 




4r7 



Table 21 
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Teachers 

The teacher responses and comments were very favorable- Most 
teachers indicated that the Program should continue without any major 
changes . 

An indirect source of information on teacher acceptance can be 
found in. the change in estimates or expectations of teachers about pupil 
performance based on responses made before and after using the SWRL 
Program . 

At the beginning of the Program, each of the 56 participating 
teachers was requested to estimate the percentage of pupils who will 
be able to achieve each of the five outcomes. At the end of the Program, 
the teacher was requested to estimate the percentage of children who are 
able to achieve each of the outcomes. 

Table 23 gives the teacher estimates of pupil performance before 
and after using the FYCSP on all Program outcomes. As the data show, 
the post-estimates were higher than the pre-estimates in all five out- 
comes, indicating that teacher perception of what children would be 
able to master had changed after use of the SWRL materials. 

Table 23 

Means of Pre- and Post-Estimates for 
Each Outcome 



OUTCQM3 


PRE -ESTIMATE 


POST-ESTIMATE 


DIFFERENCE 


I 


69.4 


82.7 


13,3 


II 


78,2 


80-3 


2.1 


III 


71*0 


74,2 


3,2 


IV 


67.6 


70*3 


2*7 


V 


51 . 9 


62,5 


10.6 


Pr .cipals 


and Supervisors 







The responses of the principals and supervisors were generally 
positive and provided a number of suggestions about changes in training 
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and monitoring procedures. The principals and supervisors also 
indicated that the vast majority of the teachers with whom they had 
contact responded favorably to the Program. 

Parent Acceptance 



Two surveys were taken of parental attitudes toward the SWRL 
Program. One questionnaire was directed specifically at parents of 
participating children in District 5. The second questionnaire was 
sent only to parents participating in the Parent-Assisted Learning 
Program. An analysis of the parent responses provided the following 
information* 

• Over 90% indicated that reading should be taught at the kinder* 
garten level . 

• Over 80% indicated a willingness to supplement school learning by 

working at home. - . 

• More than 90% felt that teaching reading in kindergarten did not 
place undue pressure on their children. 

• Virtually all parents praised the Program procedures and materials. 

Anecdotal information from the teachers and principals further substan- 
tiated parental acceptance. 



i 

I 
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Pupil Acceptance 

The Pupil Activity Preference Form was designed to give an 
indication of pupil attitudes toward the Program activities as compared 
with other kindergarten activities- The forms administered consisted 
of three pictures showing Program activities and two showing non— Program 
activities. Non-Program activities pictured are those that are typically 
found in kindergarten" class . The test booklets provided all possible 
combinations (two at a time) of five pictured activities. Two additional 
pictures were used for practice. Pictures were assigned randomly to 
the top and bottom of the pafe and the pages were randomly sequenced. 

The Activity Preference Form was presented to 160 kindergarten 
children in 20 classes in District 5. Four boys and four girls were 
selected at random from each class for testing. Children were shown 
all pairs of pictures and asked to select the activity they would most 
like to do# Forms were hand scored and results were recorded on pupil 
response sheets. Rank order of preference was determined and results 
reported in Table 24, 



* 
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Table 24 

Rank Order of Picture Preferences for 
Girls, Boys, and Total Sample 



Girls 



Parent- SWRL Storybook ....... 59 .06% 

Sharing 54.69% 

Teacher* SWRL Storybook 50.00% 

Teacher* SWRL Flashcard 47.19% 

Teacher-Number Flashcard ...» 39*38% 



N — 80 



Boys 



Sharing 59 . 69%, 

Teacher-SWRL Flashcard 52*50% 

Reliability — *68 Teacher- SI^RL Storybook 52. 19% 0 

(R 21) Parent -SWRL Storybook 49.38% 



Teacher-Number Flashcard .... 35 . 30% 



N = 80 



Total Group 



Sharing 58 . 75%, 

Parent “SWRL Storybook . . 54.2 2%, 

Teacher-SWRL Storybook . 51.06%, 

Teacher* SWRL Flashcard 50.47%, 

Teacher*Number Flashcard .... 35.47%, 



N = 160 



Gilearly the children preferred SWRL Program activities over other 
typical kindergarten instructional activities. Non- ins true tiona 1 
activities were only slightly preferred over Program activities. There 
was no evidence that reading activities were disliked by the partici- 
pants. These findings are consistent with the comments of teachers 
and SWRL observers noted in the Field Visit Logs. 



Specification of Installation Requirements 



Management Requirements 



A major objective of the installation study was the identification 
of the procedural and management requirements for effective use of the 
F±rst«=Year Communication Skills Program* During the tryout, the 
instructional management system of the Program was analyzed, including 
the instructional procedures, the instructional sequence, and perfor- 
mance requirements . 
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Instructional Management 

Precise information was required regarding the £BTRL-pre0Cribed 
procedures actually followed by teachers throughout the Program, 

Primary data sources for obtaining this information the teacher 

questionnaires j Weekly Logs, and procedural observati°h scales. Based 
on the analysis of data, the following instructional ptocedut® require- 
ments were identified: 

Daily assessment . Of particular concern to many teachers were 
the logistical problems encountered in pacing instruction fof both 
fast and slow learners. Despite a recommendation to so * f e v teachers 

were observed to assess pupil performance on a reguia^ basis* In those 
instances in which individual pupils were assessed r r^lly, te^ c hers sel- 
dom called on a representative group of pupils in tl tl class. In order 
for daily assessment to be used by all program teache? r s, a procedure 
for sampling pupils was required* The method had t D he brief* simple, 
and convenient for the teacher to use. Based on the 0 e requirements, 
special Daily Assessment Pads were designed for use during th® 1969-70 
tryout using a simple random sampling procedure- Tha pads required 
only that a teacher list the names of her pupils Qt\Qg* She then would 
assess those pupils having a blank space by their natn e * 

Procedural aids . It was observed that some teachers either failed 
to use the suggested instructional procedures or repeated difficulty in 
learning them in the initial stages of the Program, Based oil the results 
g£ the instructional task analysis, it was determined that tho Teacher's 
Manual was too bulky to use conveniently as a referet% Ce aid during 
instruction. Cards, on the other hand, were found to be easy to use 
(e * g . , the teacher could set them on her lap or by bet side during 
instruction). In addition, they could be stored with the flashcards 
and coded for easy filing. Based on these requirement s » procedural 
cards were developed for all basic instructional and clerical tasks 
for the 1969-70 tryout. 

Reinforcement procedures . A number of teachers w cre observed who 
seldom reinforced pupils for their performance in the Program* Addi- 
tional training requirements for reinforcement needed definition and 
implementation. The u se of different reinforcers was studied, such 
as the highly effective "Good Work Badges," The badges were originally 
used in the SWRL Tutorial Program, but teachers als 0 indicated their 
interest in using them as rein forcers for all pupil s lh the program. 

Based on this information, the badges were included, a® a part of the 
Program, during the 1969-70 tryout. 

I ns t actional sequence . Analysis of pupil performance data and 
instructional procedures indicated that many of the pupils who failed 
to reach criterion on the Unit Criterion Exercise Wef e not given 
remedial instruction by the teacher. It was also f Q1 .ind that In those 
instances where remedial practice was provided, teach e ts interested 
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in observing the effects of such instruction. Based on this information, 
the unit instructional sequence was redesigned to build in second in- 
struction and retesting as part of the basic instructional system. The 
revised instructional sequence consisted of the following steps. 



a. Determine skill to be taught 

b. Find Procedure Card and materials 

e. Provide group instruction 
d. Conduct dally assessment 

e- Provide additional practice, if needed 

f. Complete steps a-e for all unit skills 
2. Initial Criterion Exercise 

a. Administer Criterion Exercise 
bp Score Criterion Exercise 

c. Record scores on Class Record Sheet 

d* Award Good Work Badges to those scoring 18 or above 
e* Determine whether more than half the class scored 18 



a. Check scores for each Outcome 

b. Administer Practice Exercises to all pupils with scores 
less than 5 



a . Readminister Criterion Exercise to pupils who scored 
less than 18 on the initial Criterion Exercise 

b. Score Criterion Exercise 

c. Record score on Class Record Sheet 

d. Award Good Work Badges to pupils who scored 18 or 
above, or who had a point gain of 3 or more 



1* Initial Instruction 



or above 



3 . Second- Ins true tion 



4 - Criterion Exercise Retest 
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Program Management 



Supervisor Procedural Manuals * The major tasks of supervisors 
included that of coordinating the ordering and shipment of materials, 
orienting district office personnel, observing classroom performance, 
and consulting with teachers and 3WRL staff regarding any problems. 

Data acquired through questionnaires and interviews indicated that the 
supervisors required more detailed procedural information than that 
provided during the installation study. Based on the analysis of 
supervisor tasks, specifications were developed for a Procedural Manual, 
Log, and Monitoring Chart. The following major tasks were used as a 
basis for organizing the content of the Procedural Manual: 

1- District Staff Orientation 

2 . Teacher Training 

3. Ordering and Distributing Program Materials 

4. Acquiring Program Performance Data 

5. Monitoring Classroom Performance and Pacing 

A Program Monitoring Chart was developed to help the supervisor keep 
track of the Teacher Weekly Log submissions and classroom pacing. A 
special log was also developed to help the supervisor attend to reie* 
vant teacher behavior during classroom observations. 



Training Requirements 

In order to determine specific teacher training requirements, a 
behavioral analysis of teacher-administered instruction was performed 
by SWRL staff (Resta & Niedermeyer, 1969). A sample of teachers using 
the Program was observed. The data acquired in this study are presented 
in Table 25 and may be summarized as follows: 



P up i 1 Res p on s e 

a. Pupil Response Rate . The number of pupil responses made 
per minute in each class (Item 4) ranged from a low of 2.04 
(Teacher 0) to a high of 6.92 (Teacher E) . The average 
response rate for the nine observed classes was 4.16, which 
is only about one response every 15 seconds. The low overt 
responca rates are somewhat surprising when considering the 
fact that the teachers, cognizant of being observed, were 
presumably demonstrating their best instructional behaviors. 

b. Proportion of Choral vs. Individual Responses . Items 6 and 
8 separate the observed responses into choral and individual 
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responses, A criterion of at least two individual responses 
to every choral response was established, and it can be seen 
that only three of the nine teachers met this criterion 
(Teachers B, C, and 0), One teacher (Teacher N) did not 
elicit a single individual response during the instructional 
session , 

c , Proportion of Individual Responses . Table 26 contains the 
distribution of the number of individual responses by pupils 
in the nine observed classes. There were no classes in which 
every child responded individually at least once. The pro- 
portion of students in each class not called upon for indivi" 
dual responses ranged from a high of 1,00 to a low of ,05, 
Averaging across classes, it can be seen that 40% of the 
pupils were never called upon to respond individually during 
the observed instructional sessions, 

d. Proportion of Boy vs . Girl Responses . Item 15 of Table 2 6 
shows that teachers called on boys in proportions that closely 
matched the proportions of boys in the classes* In the seven 
classes from which these data were obtained, boys comprised 
51% of the pupils and were called on 50% of the time. 

Response Consequence 

a. Verb a 1 Re i n To r c emen t . During the observed lessons, the teachers 
praised either Individual pupils or the class as a whole an 
average of 2,53 times (Item 13). When divided by the average 
instructional time (Item 3) , this comes out to about one 
praising statement every six minutes. 

b. Teacher Confirmation . After eliciting a choral response, it 
was desirable that the teacher verbalize the correct response 
again for those pupils who may have been incorrect and could 
not distinguish what the rest of the class was saying* The 
scores on Item 7 show that teachers displayed this behavior 
only 1 6% of the time. From the scores on Item 12, it can be 
seen that only one teacher (Teacher 0) confirmed correct 
pupil responses more than 80% of the time. While Teachers 

A, B, and C (high- scoring pupils) had particularly low scores 
on this item, one could argue that their pupils needed less 
confirmation (see Items 1 and 17) than did the pupils of, for 
instance. Teacher P (low-scoring pupils). 

c. Prompting . Item 14 data indicate that all but two of the 
teachers attempted to prompt incorrect pupils. Teacher M 
prompted 20 times. Out of a total of 34 prompts attempted 
by six teachers, only six elicited the correct response. 
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Distribution of Number of Pupils Making Individual Responses 
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Teacher Training Objectives 

Several teacher training objectives were derived as a result of 
the data in Table 25* The objectives are classified below by type of 
activity and in relation to the items afi lable 25: 

& Approp riat e Instructional S timulus -Response Sequence (Item 5) . 
To distinguish between appropriate and inappropriate stimulus 
materials and response practice for skill development s given 
examples of lessons for each skill* 

• Individual Practice (Items 6 and 8) . To identify practice 
situations conforming to individual practice requirements , 
given examples* 

• Confirmation and Praise (Items 7, 12 , and 13). To identify 
appropriate confirmation and praise statements s given examples. 

• Correction (Items 10 and 11) . To identify appropriate pro- 
cedures for dealing with wrong responses and non-responses , 
given examples . 

• Prompting (Item 14). To distinguish between desirable and 
undesirable prompts, given examples of each type. 

m Response Frequency (Item 4) . To distinguish between instruc- 
tional activities which are likely or unlikely to generate 
a response rate of at least six responses per minute. 

Data provided in field visit logs, teacher questionnaires , and 
interviews , indicated that ths following additional teacher training 
objectives should be established. 

Given specific practice materials, the teacher should be able to: 

1. specify the appropriate instructional activities and materials 
to be used for the next day 

2. pronounce beginning sounds, ending sounds and blends 

3, present the visual and auditory stimuli for words. Initial and 
ending sounds, letter names, and blending 

4, identify procedures for presentation of storybooks, use of 
Animal Cards, Let ter -Naming Exercises, and Entry Skills Tests 

prepare Class Record Sheets and specify children requiring \ 
remediation I 
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6, prescribe remedial activities for specified pupils. If tutors 
are used, the teacher will be able to correctly fill out a 
Tutor Assignment Sheet 

7. fill out the Weekly log and other data forms 

The teacher should also be able to broadly describe specific 
features of the Program. 

Identification of Supervisor Training Requirements 

Because of the number and complexity of teacher training objectives, 
distirict personnel new to the Program could not be expected to carry 
heavy instructional burdens in training teachers unless far more lengthy 
and intensive training was provided for them. Such training would be 
expensive and would not guarantee high-quality teacher training. A 
safer strategy appeared to be that of developing "self-contained 11 
training materials that would enable district supervisors to manage 
the teacher training without having to assume major instructional 
responsibilities . 

In addition to the teacher training responsibility, the supervisors 
have to be familiar with Program management requirements. The following 
objectives were derived for the training of supervisors : 

1- Using SWRL materials, the supervisor will be able to plan and 

conduct both teacher training and district orientation programs. 

2. Given sample practice materials the supervisor will be able to: 

a. identify all Program materials and describe the process of 
acquiring and distributing them 

b* outline major Program events in sequence of occurrence 

c» specify methods of collecting * storing, and transmitting 
Program data forms 

d. order and control distribution of Program materials 

e. effectively monitor data acquisition process and retrieve 
missing data forms 

3* Upon completion of the Program, the supervisor should be able 
to describe the specific logistical and monitoring tasks for 
effective installation of the Program. 
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Evaluation Requirements 



The data recording tasks in this study were heaviest for teachers 5 
although efforts were made during the tryout to reduce their clerical 
tasks. Data specification and form revision requirements were identi- 
fied based on information provided by participants and changes which 
were specified for the total data management system. A summary of the 
proposed changes to the system of data collection, processing, and 
analysis is given below. 



Evaluation Sources 

The sources of data used in the study were considered quite ade- 
quate. Results indicated, however, that some of the data gathered were 
not particularly valuable for Program revision and evaluation, ' so, 
based on the suggested revisions to the training and management )ce- 
d tires some additional categories of information were suggested for 
inclusion in the full-scale installation of the FYGSP . The following 
changes in the information requirements were suggested: 

Teachers . Additional data on pupil characteristics, including 
ethnic background , previous education and other related demographic 
information were needed from the teachers , as was more precise inf or— 
ma ti on on instructional procedures, particularly the time spent teaching 
each activity and unit in one school day. Also, based on the results 
of the instructional techniques, data were needed on the use of daily 
assessment and second instruction along with estimates of the class 
time these procedures consumed. Finally, data on pupil performance 
after second instruction were needed. 

Program Supervisor , Data need to be gathered on suggested revisions 
to monitoring procedures by the supervisor. Specifically, a record of 
their classroom observation and teacher contacts is required. 

Parents. Data are required describing the use of Program materials 
outside the classroom and the possibility of utilizing such learning 
in a systematic way. More complete information on Program acceptance 
by parents is also needed. 

Instrument Evaluation 

The data indicated that extensive revisions would be required on 
the forms and measures used to collect and evaluate data in an installa- 
tion tryout. The additional data requirements outlined in the preceding 
sections pointed to the necessary revisions. On the basis of these 
specifications, the following changes in instrumentation requirements 
were suggested: 

Class Record Shee t, The additional categories of retest information, 
selection for second instruction and pupil statistics should be added 
to the Class Record Sheet . 
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Pupil Data Form . The coding for ethnic background 5 physical 
handicaps s and so on should be specified with relevant categories 
identified . 

Teacher Observation Form * Minor revisions to the form measuring 
response dispersion should be made in order to simplify the coding 
process. Revisions of the content of the Instructional Sequence 
Observation form and a simpii fication of rating procedures should be 
made , 

Weekly Log . Major revisions should be made to provide specific 
time information on initial instruction activities and second instruc- 
tion , 

Supervisor Log , A form should be developed providing information 
on (1) problems and comments made to supervisions about the Program by 
teachers and (2) a record of the supervisor's classroom observation of 
teacher instructional procedures. 

Supervisor Monitoring Chart * A form should be developed which 
would let the supervisor know at all time* the unit and activity on 
which each class is currently receiving instruction. 

Data Management and Processing Requirements 

A task directly related to instrumentation is the management of 
data collected. The study revealed a need for extensive revisions of 
the data management system in order to; 

1. reduce the time expended by teachers in preparing data 
forms 

2 * reduce the time expended by SWRL personnel in preparing the 
data for analysis and storage 

3. provide more flexible storage of the data to facilitate 
retrieval for analytic purposes 

In order to reduce the time requirements for both teachers and 
SWRL personnel in recording and preparing data for storage and analysis* 
the following revisions were made. 

Revisions to Data Management Systems 

The Glass Record Sheet and Weekly Log data management systems forms 
were revised to optical scanner format so that both teachers and SWRL 
staff need perform only a minimum of clerical tasks. 
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The transcription procedures, described earlier, were eliminated 
by adopting scanner forms. Precoding > coding., and keypunching opera- 
tions of these forms are no longer necessary. Using the revised 
systems, the forms were read directly oy the scanner equipment and the 
information was recorded on magnetic tape. Retrieval requests, as well 
as Class Record Sheets for the next unit Criterion Exercises, were 
generated by the computer, A further advantage of the system was that 
class rosters and the district, school, class, and teacher identifica- 
tion were all computer printed. The clerical time for completing forms 
was reduced and teachers are now only required to make a black mark in 
the most appropriate category. Also, errors in the transcription of 
data from manual forms to keypunched cards were eliminated. 

User evaluation forms and questionnaires, while not revised to 
scanner format, are now compatible with computer processing procedures. 
These forms have also been revised to gather data on current Program 
revisions and additions , 

At the same time, extensive revisions were made to the data 
retrieval and analysis system. As indicated, much of the analysis and 
storage of information was provided manually for the data management 
and processing requirements study. Further, the data were eventually 
stored on a time sharing system in three data bases. While the time 
sharing aspect of the system was valuable in providing immediate access 
to stored information, the particular system used had rather limited 
analytic capabilities. It was not possible, for example, to use 
information from more than one data base at a time. Moreover, the 
capacity to look at information within one data base was limited and 
precluded certain types of analysis. 

As a result , a revised data storage and analysis system was 
developed for the 1969-70 tryout. This system retains the advantage 
of a time sharing system but also provides for batch compiling of the 
information from several tape files* Such a system greatly increases 
the possibilities for analysis. 



The results of the installation requirements study provided 
information for planning and developing procedures for the subsequent 
First-Year Communication Skills Program tryout. But important questions 
needing further study were also raised related to the areas of training, 
management, and data acquisition. 



Recommendations for Additional Installation Research 
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Training 

Further development is required to handle remote training of 
teachers. The following strategy was suggested for development of 
teacher training programs for the 1969-70 installation tryout; 

L analyze FYCSP instructional and management tasks 

2* specify teacher and supervisor training objectives 

3. establish task performance criteria 

4. develop performance measures 

5. select media and develop teacher training program components 

6- try out program components (in-house) 

7* revise program components 

8. try out program in field (as part of Fall 1969 Program Tryout) 

New training concepts should also be investigated and empirical determi- 
nation should be made of such factors as: (1) the amount of training 

required for mastery of specific procedures; (2) the different options 
for introducing training within districts; and (3) the degree to which 
the training equipment can be operated manually without loss of effec- 
tiveness , 



Management 

The effectiveness of different instructional and Program management 
strategies also needs investigating. Procedures for district monitoring 
of the Program and class performance must be identified and tested if 
the Program is to be used successfully without even indirect monitoring 
by SWRL, One method of identifying these procedures is to work with a 
small group of supervisors in reviewing class pacing and performance 
and developing strategies for assisting teachers with low performing 
classes , 

Although all prototype SWRL programs have a built-in management 
system, other types of external management systems that may be used to 
increase effectiveness deserve consideration. 



Another important instructional management problem is that of 
providing pupils with appropriate levels of practice related to the 
specific Program objectives and sub-objectives . The present instruc- 
tional system has a single set of fixed-length practice materials keyed 
to outcomes. Analysis of Criterion Exercise data showed that the fixed 
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remedial treatment enabled most pupils scoring near criterion (14-17) 
to reach criterion after completing the practice exercises. Few pupils 
scoring far below criterion (Q— 11) , however ? were able to reach criterion 
after the remedial sequence. Thus 5 research is needed related both to 
the type and amount of practice required to facilitate acquisition of 
Program skills by pupils at the different performance levels. This area 
of investigation should be given high priority during subsequent try- 
outs * 



Data Acquisition 

Of particular importance in data acquisition is the need for 
defining and gathering longitudinal data on the Program participants. 
Such information is of special import in establishing an empirical basis 
for the Program criterion level. By looking at the performance levels 
of SWRL pupils on subsequent measures of reading and language arts and 
relating these to their earlier performance on the SWRL FYCSP, the 
necessary criterion level of SWRL performance needed to reach acceptable 
levels of later performance can be established empirically. It is also 
conceivable that longitudinal data on other Program objectives such as 
pupil acceptance of reading could be gathered and used to determine the 
long-term effect on such behavior* 

Another important topic in this area is item and pupil sampling. 
With the increasing size of both the tryout populations and the amount 
of information required 3 such sampling schemes will be necessary means 
of keeping the quantity of information gathered at reasonable levels 
while providing data on all Program objectives. Student sampling was 
used on a limited basis during the current requirements study primarily 
to estimate class performance on the midterm final tests. Additional 
research on the utility of various sampling procedures should provide 
the information needed to apply sampling methods to other aspects of 
data gathering. 

A third area of data acquisition requiring investigation deals with 
the effectiveness of the teacher and supervisor training. For the 
requirements study the only information obtained on training effective- 
ness was through teacher and supervisor questionnaires and classroom 
observations of teachers. While this information provides gross 
indicators of the requirements , more precise data will be needed in the 
future on specific training objectives. One means by which such data 
might be gathered is through a series of in-house training sessions 
followed by specific evaluation procedures. Such studies should provide 
more precise information on each of the training objectives as w^ll as 
on general strategies to follow in teacher training. 




